A private web server must not respond to requests from public search engines.

Overview

Finding ID	Version	Rule ID	IA Controls	Severity
V-2260	WG310 A22	SV-33028r1_rule	ECLP-1	Low

Description
Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web site content. In turn, these search engines make the content they obtain and catalog available to any public web user. Such information in the public domain defeats the purpose of a Limited or Certificate-based web server, provides information to those not authorized access to the web site, and could provide clues of the site’s architecture to malicious parties.

Description

Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web site content. In turn, these search engines make the content they obtain and catalog available to any public web user. Such information in the public domain defeats the purpose of a Limited or Certificate-based web server, provides information to those not authorized access to the web site, and could provide clues of the site’s architecture to malicious parties.

STIG	Date
APACHE SITE 2.0 for Unix	2011-12-12

Details

Check Text ( C-33711r1_chk )
Note: Additional restrictions can be stated in the robots.txt file. The below example will disallow access to all directories in the document root directory. If the robots.txt doesn’t have “User-agent” or “Disallow” statements set, this is a finding. Query the Web Administrator to determine what type of restriction from public search engines is in place. If robots.txt files are used, locate the Apache httpd.conf file. If unable to locate the file, perform a search of the system to find the location of the file. Open the httpd.conf file with an editor such as Notepad, and search for the following uncommented directives : DocumentRoot & Alias Navigate to the location(s) specified in the Include statement(s), and review each file for the following uncommented directives: DocumentRoot & Alias At the top level of the directories identified after the enabled DocumentRoot & Alias directives, a “robots.txt” file should exist. If the file does not exist, this is a finding. For any robots.txt files that do exist, open them in a text editor (i.e. notepad) and ensure they contain the following text: User-agent: * Disallow: / If no means of restriction is in place (e.g. userid and password, domain or IP restriction, PKI), or a robots.txt file is not in use, this is finding.

Check Text ( C-33711r1_chk )

Note: Additional restrictions can be stated in the robots.txt file. The below example will disallow access to all directories in the document root directory. If the robots.txt doesn’t have “User-agent” or “Disallow” statements set, this is a finding.
Query the Web Administrator to determine what type of restriction from public search engines is in place.

If robots.txt files are used, locate the Apache httpd.conf file.

If unable to locate the file, perform a search of the system to find the location of the file.

Open the httpd.conf file with an editor such as Notepad, and search for the following uncommented directives : DocumentRoot & Alias

Navigate to the location(s) specified in the Include statement(s), and review each file for the following uncommented directives: DocumentRoot & Alias

At the top level of the directories identified after the enabled DocumentRoot & Alias directives, a “robots.txt” file should exist. If the file does not exist, this is a finding. For any robots.txt files that do exist, open them in a text editor (i.e. notepad) and ensure they contain the following text:

User-agent: *
Disallow: /

If no means of restriction is in place (e.g. userid and password, domain or IP restriction, PKI), or a robots.txt file is not in use, this is finding.

Fix Text (F-29343r1_fix)
Establish a means to restrict search engines on the private web site.